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ABSTRACT 

We  describe  a  framework  for  analyzing  simulation  output 
in  order  to  find  solutions  that  will  work  well  after  imple¬ 
mentation.  We  show  how  the  use  of  a  loss  function  that 
incorporates  both  system  mean  and  system  variability  can  be 
used  to  efficiently  and  effectively  carry  out  system  optimiza¬ 
tion  and  improvement  efforts.  For  models  whose  behavior 
depends  on  quantitative  factors,  we  illustrate  how  robust 
design  can  be  accomplished  by  using  simple  experimental 
designs  in  conjunction  with  response-surface  metamodels. 
The  results  can  yield  new  insights  into  system  behavior, 
and  may  lead  to  recommended  system  configurations  that 
differ  substantially  from  those  selected  by  analysis  solely 
on  the  basis  of  mean  response.  We  assume  a  knowledge 
base  at  the  level  of  Chapter  12  of  Simulation  Modeling  and 
Analysis  (Law  and  Kelton  2000)  but  will  review  essential 
elements  and  distribute  illustrative  examples  at  the  session. 

1  INTRODUCTION 

What  is  robust  design?  It  is  a  system  optimization  and  im¬ 
provement  process  that  springs  from  the  view  that  a  system 
should  not  be  evaluated  on  the  basis  of  mean  performance 
alone.  In  addition  to  exhibiting  an  acceptable  mean  perfor¬ 
mance,  a  “good”  system  must  be  relatively  insensitive  to 
uncontrollable  sources  of  variation  present  in  the  system’s 
environment. 

Some  notation  and  terminology  will  be  necessary  before 
the  robust  design  process  and  its  benefits  can  be  fully 
discussed.  However,  it  is  important  to  state  up  front  that 
the  purpose  of  robust  design  is  to  lead  to  better  decisions: 
better  in  terms  of  implementation,  better  in  terms  of  the 
level  and  consistency  of  performance,  better  in  terms  of  cost, 
and  better  in  terms  of  insights  into  the  drivers  of  system 
performance. 

The  robust  design  approach  originated  in  quality  plan¬ 
ning  and  engineering  product  design  activities  (Taguchi  and 


Wu  1980;  Taguchi  1986  1987).  Taguchi  found  that  it  was 
often  more  costly  to  control  causes  of  manufacturing  vari¬ 
ation  than  to  make  a  process  insensitive  to  these  variations, 
and  through  the  use  of  simple  experimental  designs  and  loss 
functions  it  was  often  possible  to  greatly  improve  product 
performance  by  “building  in”  the  quality.  Taguchi’s  philos¬ 
ophy  and  strategy  have  been  widely  praised  in  both  the  ap¬ 
plied  statistics  and  manufacturing  communities,  (Pignatiello 
and  Ramberg  1991)  but  many  of  the  methods  and  tactics  he 
advocates  are  often  controversial  (Box  1988,  Ramberg,  Pig¬ 
natiello  and  Sanchez  1991,  Nair  et  al.  1992).  The  approach 
described  in  this  paper  (see  also  Sanchez  et  al.  1996  and 
1998,  Ramberg  et  al.  1991)  combines  Taguchi’s  strategy  and 
response-surface  metamodeling  techniques.  The  additional 
insights  that  can  be  gained  make  this  approach  particularly 
beneficial  when  analyzing  simulations  of  complex  systems. 

In  the  simulation  context,  robust  design  can  be  viewed 
from  two  slightly  different  perspectives.  One  view  is  that 
simulation  is  used  primarily  as  a  surrogate  for  a  real  system, 
because  of  the  cost  and  time  required  to  make  and  observe 
changes  in  a  real  system.  From  this  perspective,  the  to¬ 
tal  time  required  to  perform  a  robust  design  experiment  is 
greatly  reduced,  but  the  designs  and  analyses  used  are  the 
same  as  those  that  would  be  applied  to  a  physical  system 
if  cost  and  time  permitted.  Applications  have  included  the 
product  designers’  uses  of  computer  models  for  experimen¬ 
tation  in  place  of  physical  prototypes,  particularly  in  the 
semiconductor  industry  (Sacks  et  al.  1989,  Welch  et  al. 
1990).  These  experiments  have  typically  involved  Monte 
Carlo  simulation,  although  clearly  robustness  can  be  used  as 
a  criteria  for  evaluating  discrete-event  simulation  systems  as 
well.  Those  who  use  simulation  to  study  systems  primarily 
because  of  the  difficulty  of  experimenting  on  the  real  sys¬ 
tem  may  realize  the  benefits  of  improved  performance  and 
decreased  cost  cited  by  many  manufacturers  if  they  decide 
to  evaluate  performance  in  terms  of  robustness. 

A  larger  view  of  the  simulation  process  is  also  possible. 
A  simulation  model  is  constructed  assuming  a  variety  of 
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system  inputs  (e.g.,  distributional  forms  and  characteristics, 
simplifying  assumptions,  level  of  detail)  which  are  unlikely 
to  be  completely  accurate.  Model  verification  and  validation 
are  important  issues  in  the  field,  as  is  simulation  sensitivity 
analysis.  From  this  perspective,  one  can  view  robust  design 
as  a  process  of  simulation  optimization,  where  the  “best” 
answer  is  not  overly  sensitive  to  small  changes  in  the  system 
inputs.  If  robust  systems  are  identified,  then  the  actual  results 
are  more  likely  to  conform  to  the  anticipated  results  after 
implementation. 

The  robustness  criteria  can  be  applied  to  rank  a  discrete 
number  of  alternatives,  which  result  from  changing  the  set¬ 
tings  of  some  or  all  of  the  inputs  to  the  simulation  model 
(or  system).  Alternatively,  if  some  or  all  of  the  input  factors 
are  quantitative,  one  can  construct  metamodels  of  the  simu¬ 
lation  which  describe  how  the  system  performance  varies  as 
a  function  of  the  input  factors.  There  are  many  approaches 
to  metamodeling,  but  response  surface  metamodels  work 
well  in  the  robust  design  context.  Metamodels  provide 
much  more  information  about  the  underlying  system  than 
haphazard  investigation  of  a  few  alternatives.  Thus,  if  the 
goal  of  the  analyst  is  to  optimize  or  improve  the  model’s 
performance,  and  flexibility  exists  in  the  settings  of  the  de¬ 
cision  factor  levels  (as  in  prospective  studies),  then  building 
a  metamodel  is  appropriate.  The  actual  number  of  config¬ 
urations  studied,  and  the  form  (linear,  quadratic,  etc.)  of 
the  resulting  metamodel  are  dependent  on  the  experimental 
design  chosen.  Note  that  first-order  models  may  not  suffice 
for  complex  discrete-event  simulations:  performance  is  of¬ 
ten  highly  nonlinear,  even  over  a  relatively  restricted  range 
of  factor  settings. 

The  construction  of  metamodels  is  facilitated  by  the 
use  of  experimental  design  techniques.  Simulation  analysts 
have  the  luxury  of  controlling  all  inputs  to  the  simulation 
(including  random  number  seeds,  etc.):  this  means  they 
have  more  flexibility  in  designing  the  experiment,  and  more 
opportunities  for  exploiting  the  additional  degree  of  control, 
than  do  those  experimenting  directly  on  real  systems. 

In  this  tutorial  our  focus  is  on  the  robust  design  process 
for  discrete-event  simulation  experiments.  We  begin  with 
a  discussion  of  the  terminology  used  in  the  robust  design 
approach.  We  then  discuss  tactical  issues,  such  as  appropri¬ 
ate  experimental  designs,  metamodel  construction,  robust 
design  identification,  analysis.  We  conclude  with  a  summary 
of  the  benefits  of  a  robust  design  approach,  emphasizing  the 
types  insights  into  system  behavior  that  can  be  achieved. 

2  NOTATION  AND  TERMINOLOGY 

In  this  section,  we  present  notation  and  discuss  the  termi¬ 
nology  related  to  the  robust  design  process. 


2.1  Factor  Classification 

In  systems  where  stochastic  variation  is  present,  the  response 
exhibits  random  fluctuation  or  variation.  In  order  to  achieve 
systems  or  products  for  which  the  variation  around  the  target 
value  is  low,  several  steps  are  necessary.  First,  one  must 
identify  factors  in  the  system  which  are  anticipated  to  affect 
the  system  response.  Factors  are  classified  as  decision 
factors,  noise  factors,  or  artificial  factors. 

The  decision  factors  are  those  which  are  controllable 
in  the  real  world  setting  modeled  by  the  simulation.  Noise 
factors  are  not  easily  controllable  or  controllable  only  at 
great  expense  in  the  real-world  setting.  Noise  factors  include 
sources  of  variation  within  the  real-world  system  (i.e.,  within 
a  manufacturing  plant)  as  well  as  exogenous  factors  (such 
as  customer  and  supplier  characteristics).  Finally,  artificial 
factors  are  those  simulation-specific  variables  such  as  the 
initial  state  of  the  system,  the  warm-up  period  (truncation 
point),  termination  conditions  (run  duration),  and  random 
number  stream(s)  (seed,  antithetic  switch). 

The  distinction  between  decision  factors  and  noise  fac¬ 
tors  is  often  recognized  in  simulation  experiments,  but  rarely 
used  to  develop  the  experimental  design  or  affect  the  anal¬ 
ysis  of  the  simulation  results.  However,  as  we  discuss  in 
Section  3.1,  the  classification  is  important.  It  is  necessary 
for  determining  system  robustness,  and  also  presents  an 
opportunity  for  reducing  the  number  of  runs  required  by 
concentrating  sampling  efforts  on  assessing  decision  factor 
effects.  This  additional  layer  of  control  made  possible  by 
the  artificial  factors  can  also  be  exploited  in  the  experimental 
design  (Schruben  et  al.  1992).  This  is  not  new — it  is  the 
basis  of  many  variance  reallocation  techniques. 

2.2  Performance  Evaluation 

The  analyst  begins  by  specifying  some  performance  char¬ 
acteristic  of  special  interest,  and  an  associated  target  value 
r.  Common  measures  are  related  to  system  throughput 
or  system  states,  such  as  the  waiting  time  or  number  of 
customers  in  queueing  systems,  although  cost  could  also 
be  used  as  a  performance  measure.  In  general,  the  pattern 
of  the  performance  characteristic’s  fluctuation  around  the 
target  value  will  differ  across  these  configurations.  The  cost 
of  this  fluctuation  must  be  measured  in  order  to  optimize 
or  improve  the  system.  Since  end-users  will  incur  costs  if 
system  performance  deviates  from  the  target,  the  evaluation 
criterion  is  often  philosophically  referred  to  as  the  loss  to 
society,  or  the  long  term  business  loss. 

An  ideal  configuration  would  result  in  the  performance 
characteristic’s  mean  equal  to  r  and  its  variance  equal  to 
zero.  Thus,  a  numerical  method  for  trading  off  performance 
mean  and  variability  is  needed. 

We  utilize  a  quadratic  loss  function,  which  (in  many 
cases)  is  a  reasonable  surrogate  for  the  ‘true’  underlying 
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loss  function  which  may  be  difficult  or  impossible  to  spec¬ 
ify  exactly.  Let  x  and  T(x)  denote  a  vector  of  decision 
factor  settings  and  the  associated  performance  characteristic 
respectively,  and  let  Q  denote  the  noise  factor  space.  Then, 
assuming  that  no  loss  is  incurred  when  Y  (x)  achieves  the 
ideal  state  (r),  the  quadratic  loss  function  can  be  written 
as: 

(■  (Y  (x))  =  c[Y  (x)  —  r]2  (1) 

where  the  scaling  constant  c  can  be  used  to  convert  losses 
into  monetary  units  to  facilitate  comparisons  of  systems 
with  different  capital  costs.  It  follows  from  equation  (1) 
that  the  expected  loss  associated  with  configuration  x  is 

E{loss )  =  c  | '  o2  +  (ji  —  r)2J  .  (2) 

This  loss  function  has  several  nice  properties.  It  pe¬ 
nalizes  small  deviations  from  r  only  slightly,  yet  assesses  a 
large  penalty  for  responses  far  from  the  target.  The  expected 
loss  is  similar  to  a  mean-squared-error  loss,  thus  it  has  many 
desirable  mathematical  properties  as  well.  However,  other 
loss  functions  can  be  used  if  the  true  loss  function  is  not 
approximated  well  by  a  quadratic. 

While  conceptually  straightforward,  the  use  of  a  loss 
function  to  incorporate  system  variability  into  the  perfor¬ 
mance  evaluation  represented  a  major  shift  in  perspective 
within  the  manufacturing  community.  No  longer  was  it  ac¬ 
ceptable  to  think  about  optimizing  mean  performance  with¬ 
out  regard  to  performance  variability:  a  “good”  product  was 
also  robust.  The  quantification  of  robustness,  instead  of  the 
0/1  loss  function  often  implicitly  used  to  represent  products 
which  were  within/outside  specification  limits,  also  provided 
impetus  to  management  and  manufacturing  to  continually 
improve  product  quality.  We  believe  that  for  many  types  of 
applications,  a  similar  change  in  perspective  should  occur 
within  the  discrete-event  simulation  community.  If  simula¬ 
tion  is  being  used  to  identify  “good”  systems  (e.g.,  plant 
layouts,  scheduling  and  control  mechanisms),  where  vari¬ 
ability  is  not  constant  across  alternative  system  designs,  then 
a  loss  function  such  as  that  in  equation  (2)  is  a  better  descrip¬ 
tor  of  the  system’s  desirability  than  the  average  (steady-state) 
performance.  For  example,  a  single-server  queue  will  have 
the  same  mean  waiting  time  for  customers/jobs  under  the 
FIFO  and  LIFO  queue  disciplines,  but  the  variability  is  quite 
different.  In  general,  the  configuration  with  the  best  mean 
need  not  be  associated  with  the  lowest  loss. 

3  EXPERIMENTAL  DESIGNS 

Choosing  an  experimental  design  means  specifying  the  lev¬ 
els  of  all  decision  factors,  noise  factors,  and  artificial  factors 
for  each  run  of  the  simulation.  An  appropriate  total  sample 
size  must  also  be  determined.  In  order  to  evaluate  the  ex¬ 


pected  response  variability  across  the  noise  space,  a  crossed 
decision x  noise  factor  plan  can  be  used.  This  means  that 
the  same  experimental  plan  for  the  noise  factors  is  used  for 
each  run  of  the  decision  factor  plan. 

3.1  Basic  Plans 

Complete  and  fractional  factorials  are  often  used.  Among 
these,  two-level  designs  are  popular  choices  because  of 
their  simplicity  and  efficiency.  They  permit  the  evaluation 
of  the  linear  decision  factor  effects,  as  well  as  interaction 
or  synergistic  effects. 

For  a  two-level  factorial  or  fractional  factorial  experi¬ 
ment  involving  k  decision  factors,  the  factor  levels  should 
be  chosen  to  cover  the  range  of  interest.  For  noise  factor 
plans,  the  levels  should  be  chosen  so  that  the  mean  and 
variance  of  the  two-point  sampling  distribution  are  equal 
to  the  mean  and  variance  of  the  underlying  distribution.  In 
the  case  of  two-level  sampling  of  continuous  factors  (or 
discrete  factors  whose  distributions  can  be  closely  approxi¬ 
mated  by  continuous  distributions),  this  corresponds  to  one 
standard  deviation  below  and  one  standard  deviation  above 
the  mean.  In  the  case  of  equally  likely  Bernoulli  outcomes, 
this  corresponds  to  the  two  factor  levels.  For  discrete  dis¬ 
tributions  where  [i  ±  a  does  not  yield  valid  factor  levels, 
the  outcomes  can  be  sampled  (approximately)  proportional 
to  their  probability  of  occurrence.  If  the  factor  is  a  mean 
estimated  from  data,  then  the  upper  and  lower  bounds  of  a 
95%  confidence  interval  ean  be  used  (Wild  and  Pignatiello 
1991). 

Other  orthogonal  designs  have  been  advocated  for  re¬ 
sponse  surface  metamodeling.  For  example,  one  might 
want  to  minimize  the  bias  or  mean-squared  error  of  the 
regression  coefficients  (Donahue,  Houck  and  Myers  1992). 
Central  composite  designs  are  good  for  fitting  second-order 
metamodels.  These  designs  are  discussed  for  simulation 
experiments  by  Tew  (1992)  or  Hood  and  Welch  (1993); 
experimental  design  texts  such  as  Box  and  Draper  (1987), 
Box,  Hunter,  and  Hunter  (1988)  or  Montgomery  (1991) 
contain  details  and  alternative  designs.  Two-level  plans  are 
not  sufficient  if  quadratic  effects  are  anticipated. 

3.2  Artificial  Factor  Plans 

A  well  established  field  in  simulation  is  that  of  variance 
reallocation  (or  variance  reduction),  where  researchers  have 
established  methods  of  reducing  the  variance  of  the  esti¬ 
mators  of  mean  responses  in  order  to  increase  power  for 
hypothesis  testing  purposes.  Unequal  response  variance  at 
different  system  configurations  is  recognized  as  pervasive. 
It  often  influences  the  experiment  design  and  analysis  (e.g., 
varying  run  lengths  for  different  system  alternatives),  but 
has  rarely  been  incorporated  into  the  system  evaluation. 
In  the  robust  design  context,  variance  reallocation  schemes 
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hold  promise  for  further  increasing  the  efficiency  of  ex¬ 
perimentation.  Rather  than  using  all  independent  random 
number  streams,  one  can  use  a  common/antithetic  sampling 
strategy  (Schruben  and  Margolin  1978,  Tew  and  Wilson 
1991,  1994).  This  reallocates  variance  among  the  coeffi¬ 
cient  estimates.  The  implications  for  robust  design  are  that 
the  artificial  factor  plan  should  be  chosen  in  order  to  induce 
correlations  which  reallocate  variance  from  the  interesting 
terms  (decision  factors)  to  the  uninteresting  terms  (noise 
factors)  (Schruben  et  al.  1992).  The  artificial  factor  plan  is 
typically  embedded  in  the  noise  factor  plan,  e.g.,  through  the 
choice  of  random  number  streams  used  during  a  simulation 
run. 

3.3  Frequency  Domain  Plans 

If  the  number  of  noise  factors  is  large,  even  a  saturated 
factorial  plan  for  the  noise  factors  may  result  in  an  unwieldy 
experimental  design  after  crossing  it  with  the  decision  factor 
plan.  One  way  to  cut  down  the  size  of  the  experiment  is 
to  first  screen  the  noise  factors  and  then  employ  a  highly 
fractionated  factorial  design.  Another  efficient  way  to  collect 
the  data  is  to  oscillate  each  noise  factor  sinusoidally  within  a 
simulation  run  at  unique,  carefully  selected  frequency.  This 
allows  examination  of  the  system  across  a  range  of  noise 
factor  combinations  without  a  prohibitively  large  number  of 
runs  (Moeeni  et  al.  1997).  Such  oscillation  forms  the  basis 
of  frequency  domain  experimentation  in  the  simulation  field 
(Schruben  and  Cogliano,  1987;  Sanchez  and  Buss,  1987), 
although  the  analysis  differs.  Indexing  by  time,  rather  than 
by  entity,  is  recommended  (Mitra  and  Park,  1991). 

For  variance  attribution,  the  analyst  is  interested  in 
determining  what  portions  of  the  total  system  variability  can 
be  attributed  to  the  noise  factors,  and  a  frequency  domain 
approach  could  be  used  for  factor  screening  purposes.  For 
robust  design,  we  are  interested  in  what  the  performance 
variability  is  at  a  particular  decision  factor  configuration: 
the  fact  that  noise  factors  are  varying  across  the  noise 
space  is  important,  while  estimates  of  their  specific  effects 
are  not.  In  both  cases,  care  should  be  taken  to  select 
driving  frequencies  which  will  not  result  in  confounding 
and  to  choose  frequencies  resulting  in  cycles  sufficiently 
long  to  affect  the  system  response  (Jacobson,  Buss  and 
Schruben  1991).  Discrete  factors  can  be  handled  either  by 
oscillating  their  probabilities  of  realizing  particular  levels, 
or  by  discretizing  the  sinusoidal  function  (Sanchez  and 
Sanchez,  1991). 

3.4  Correlated  Factor  Plans 


factors  were  held  at  their  high  levels.  If  this  situation  was 
unlikely  to  occur  in  practice  because  of  correlation  among 
the  variables,  then  a  sampling  scheme  which  made  use 
of  the  underlying  dependence  structure  would  seem  more 
appropriate.  If  the  noise  factors  are  normally  distributed,  the 
analyst  can  sample  at  axial  points  on  the  elliptical  contours 
of  the  joint  distributions  (Sanchez  1994b,  Sanchez,  Smith 
and  Lawrence  1996,  2000). 

3.5  Combined  Array  Plans 

In  some  circumstances,  a  crossed  decision  x  noise  factor 
plan  may  not  be  the  most  efficient  in  terms  of  the  total 
number  of  observations  (runs)  required.  An  alternative  is 
a  combined  plan,  where  a  single  design  matrix  (such  as  a 
factorial)  is  used  with  columns  divided  among  decision  and 
noise  factors.  As  Myers,  Khuri  and  Vining  (1992)  suggest, 
this  can  be  used  if  one  can  specify  a  priori  which  of  the 
many  possible  interaction  terms  are  potentially  important. 
It  may  mean  that  the  experiment  can  be  conducted  using  a 
smaller  total  number  of  simulation  runs  than  a  crossed  plan 
would  require. 

4  RESPONSE  SURFACE  METAMODELS 


The  response  Y  is  a  random  function  of  the  decision  fac¬ 
tors  {X,},  the  noise  factors  { VUy } ,  the  artificial  factors  {A, t}, 
and  the  inherent  variability  of  the  system.  The  form  of 
the  metamodels  fit  to  the  simulation  outputs,  and  the  meta¬ 
model  uses,  differ  between  the  robust  design  and  variance 
attribution  stages. 

4.1  Metamodels  for  Robust  Design 


For  this  analysis,  we  seek  to  characterize  the  system  be¬ 
havior  as  a  function  of  the  decision  factors  alone.  For 
every  combination  of  decision  factor  configuration  i  and 
noise  factor  configuration  j,  we  first  compute  (after  suitable 
truncation  to  remove  initialization  bias)  the  sample  average 
Y ij  and  sample  variance  sr.  for  the  run.  Then  we  compute 
summary  measures  across  the  noise  space  for  each  decision 
factor  configuration  i: 
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If  the  noise  factors  are  correlated  in  the  real  world  system,  it 
might  be  that  a  factorial  or  fractional  factorial  design  could 
not  be  conducted  over  the  entire  range  of  interest.  For 
example,  a  queueing  system  might  be  unstable  if  all  noise 


where  nw  is  the  number  of  points  in  the  noise  factor  plan. 
Regression  is  used  to  fit  two  initial  metamodels:  one  for  the 
performance  mean,  and  one  for  the  performance  variability 
(Sanchez  et  al.  1993,  1998;  see  also  Vining  and  Myers, 
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1990).  The  terms  in  the  initial  metamodels  depend  on  the 
experimental  design  used.  For  discrete-event  simulation 
experiments  we  recommend  a  design  which  allows  for  fitting 
at  least  a  quadratic  effect.  We  obtain  models  such  as 

[x  «  PQ  +  PiX1  +  ...  +  pkXk  +  PiaXiX2  (3) 

+  . . .  +  Pk-\,kXk-\  %k  +  quadratic 

log O2)  «  yo  +  Y\X\  +  . . .  +  nXk  (4) 

+Vl, 2X1X2  +  .  .  .  +  Yk-\,kXk-lXk 

+  quadratic 

The  logarithmic  transformation  is  used  for  stability  purposes. 
If  the  quadratic  is  an  important  term  in  either  metamodel, 
further  experimentation  is  needed  to  determine  the  decision 
factor(s)  from  which  it  arises. 

4.2  Metamodels  for  Variance  Attribution 

Metamodel  construction  is  slightly  Different  for  variance 
attribution  purposes  since  all  factors  are  treated  as  noise 
factors  and  we  assume  that  the  factor  ranges  are  sufficiently 
small  that  a  linear  metamodels  suffice.  If  we  fit  models  of 
the  response  mean  and  standard  deviation,  then  we  obtain 

M  «  Po  +  PiWi  +  ...j}kWk, 

W 

s  «  YO  +  Yl.Yj  WJ 
7=1 

where  the  {ftj}  and  {yj}  are  the  least-squares  regression 
coefficients.  By  treating  these  coefficients  as  fixed,  the 
overall  variance  can  be  approximated  by 

W 

»  Yo  +  +  Yj)Vn(Wj).  (5) 

7=1 

5  ANALYZING  THE  RESULTS 

The  initial  metamodels  constructed  for  either  robust  design 
or  variance  attribution  should  be  assessed  and  may  need  to  be 
refined.  The  experimental  plans  are  typically  unreplicated 
because  of  the  cost  of  experimentation.  This  means  that  the 
analyst  may  have  only  a  single  degree  of  freedom  for  error  in 
the  initial  regression  metamodels,  so  heavy  reliance  should 
not  be  placed  on  the  raw  /7-values  or  r-values.  An  option 
offered  by  many  statistical  packages  (or  which  can  be  done 
manually)  is  a  normal  probability  plot,  which  can  be  used  to 
graphically  assess  whether  or  not  any  effects  larger  than  the 
noise  thresh-hold  of  the  the  experiment.  Normal  probability 
plots  work  well  when  15  or  more  factor  or  interaction  effects 
are  estimated.  If  the  regression  metamodels  can  be  simplified 


by  eliminating  unimportant  terms,  then  pooling  increases 
the  degrees  of  freedom  for  the  error  estimate  and  allows 
formal  tests  of  the  statistical  significance  of  the  remaining 
metamodel  coefficients. 

5.1  Robust  Design  Analysis 

The  information  resulting  from  the  robust  design  metamod¬ 
els  of  equations  (3)  and  (4)  can  easily  be  combined  using  the 
quadratic  loss  function  (equation  (2))  to  identify  robust  con¬ 
figurations.  The  metamodels  themselves  provide  detailed 
information  regarding  the  system  performance:  they  indi¬ 
cate  which  decision  factors  affect  the  mean,  which  affect  the 
variance,  and  which  influence  both  aspects  of  performance. 

For  many  simulation  models,  the  presence  of  interac¬ 
tion  terms  and  the  relationship  between  the  mean  and  the 
variability  of  the  performance  characteristic  make  it  difficult 
to  achieve  the  target  value  with  the  most  robust  product 
design.  In  these  cases,  contour  plots  may  be  useful  for 
selecting  candidate  product  designs.  For  example,  one  can 
first  use  the  mean  metamodel  to  identify  several  configu¬ 
rations  for  which  the  average  performance  characteristic  is 
on  target,  and  then  use  the  the  metamodel  of  log  (a2  )  to 
select  a  configuration  which  is  fairly  robust. 

Often  the  results  suggest  configurations  that  were  not 
among  those  initially  tested.  In  such  cases,  further  experi¬ 
mentation  is  beneficial  in  order  to  confirm  the  performance 
characteristic’s  behavior  before  committing  to  a  particular 
configuration.  However,  the  secondary  experiment  may  be 
much  smaller  than  the  initial  experiment  if  several  of  the 
decision  factors  do  not  appear  in  the  revised  metamodels: 
they  can  be  set  at  their  most  economical  levels  and  screened 
from  further  experimentation. 

We  emphasize  that  the  decision  in  the  robust  design 
framework  can  be  very  different  than  that  made  on  the  basis 
of  mean  performance  alone.  Even  for  queueing  systems, 
where  performance  mean  and  variability  tend  to  have  high 
positive  correlation,  complex  interactions  among  decision 
factors  may  affect  that  relationship.  One  example  (Sanchez, 
Ramberg  and  Sanchez  1998)  that  will  be  illustrated  in  detail 
during  the  presentation  concerns  a  job-shop  simulation.  The 
job-shop  has  three  products,  five  machine  groups,  and  vary¬ 
ing  processing  time  distributions,  product  mix  percentages, 
etc.  Experimentation  showed  that  two  configurations  could 
have  the  same  mean  response  to  two  decimal  places,  yet 
variances  which  differed  by  over  a  factor  of  two.  The  config¬ 
urations  corresponding  to  the  best  means  were  dramatically 
inferior  to  the  low  loss  designs:  one  job  shop  configuration 
which  was  among  the  best  in  terms  of  mean  performance 
had  a  36%  higher  loss  then  the  low-loss  configuration,  yet 
it  used  more  machines. 

The  robust  design  philosophy  and  joint  metamodeling 
approach  have  a  synergistic  relationship:  they  typically 
provide  the  analyst  with  more  information  than  would  result 
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from  either  a  loss  comparison  of  only  the  configurations 
tested,  or  from  a  single  metamodel  which  directly  measures 
system  loss  or  cost.  In  the  latter  case,  if  a  metamodel  shows 
that  expected  loss  decreases  as  factor  X  increases,  the  root 
cause  remains  unknown.  Perhaps  the  response  mean  is 
closer  to  the  target.  Perhaps  the  response  variance  is  smaller. 
It  could  be  that  both  the  mean  and  variance  improve,  or 
that  an  improvement  in  one  aspect  is  partially  offset  by  a 
degradation  in  the  other.  However,  separate  construction  of 
metamodels  for  the  system  mean  and  variability  facilitate 
the  identification  of  new  designs  which  may  be  even  better 
than  those  considered  in  the  experimental  framework. 

5.2  Variance  Attribution  Analysis 

Several  types  of  questions  can  be  addressed.  First,  one 
can  assess  the  overall  mean  and  variability  for  a  particular 
configuration,  e.g.,  that  chosen  at  the  end  of  the  robust  design 
stage.  If  the  decision  factors  can  be  perfectly  controlled  at 
their  chosen  settings,  then  the  overall  mean  and  variance 
can  just  be  estimated  by  the  robust  design  metamodels. 
However,  if  variation  in  the  decision  factors’  settings  around 
their  nominal  values  is  anticipated,  an  additional  experiment 
will  provide  a  better  picture  of  the  system’s  capabilities. 

Other  questions  concern  the  relative  effects  of  the  noise 
factors.  The  term  +  j/?)Var(W;)  in  equation  (5)  is 
called  the  transmitted  variance  for  noise  factor  Wj.  This 
indicates  the  amount  of  variability  in  the  noise  factor  which  is 
passed  along  to  variance  in  the  response.  Depending  on  the 
magnitudes  of  fij  and  yj,  variation  in  Wj  can  be  amplified 
or  dampened  by  the  system.  The  term  in  equation  (5) 
is  called  the  inherent  variance :  it  is  the  smallest  variance 
achievable  if  all  noise  factors  investigated  in  the  experiment 
have  variances  driven  to  zero.  (For  Monte  Carlo  simulations, 
all  randomness  has  been  removed  during  experimentation 
and  every  y,  can  be  replaced  by  zero.) 

Once  transmitted  variances  have  been  computed  for  all 
noise  factors,  the  relative  importance  of  these  sources  of 
variation  on  the  output  is  apparent.  This  information  can 
be  used  to  evaluate  proposed  changes  to  the  system.  For 
example,  is  it  cost-effective  to  pay  more  for  raw  materi¬ 
als,  machine  maintenance,  or  training  in  order  to  improve 
the  consistency  of  these  factors?  Alternatively,  is  it  possi¬ 
ble  to  relax  controls  slightly  and  allow  more  variation  in 
the  inputs  without  adversely  affecting  the  system  behav¬ 
ior?  If  the  standard  deviation  of  a  noise  factor  Wj  can 
be  reduced  by  a  factor  of  a/  without  affecting  the  mean 
performance,  then  its  transmitted  variance  is  reduced  by  a 
factor  of  aj  and  the  overall  system  variability  is  reduced 
by  the  amount  (1  —  aj)V'dr(W  j).  The  conversion  constant 
c  from  equation  (2)  can  be  used  to  express  the  overall  per¬ 
formance  change  in  dollars.  A  comparison  with  the  cost  of 
implementing  the  proposed  change  then  shows  whether  or 
not  such  implementation  would  further  improve  the  system 


performance.  If  changes  in  the  noise  factor  variance  also 
affect  the  system  mean,  then  both  the  mean  and  variance 
components  should  be  included  in  the  cost  assessment  via 
equation  (2). 

Variance  attribution  can  also  aid  in  simulation  modeling 
and  validation.  If  the  distributional  characteristics  used  to 
generate  random  components  of  the  simulation  model  are 
themselves  estimates,  and  if  the  system  is  highly  sensitive  to 
those  characteristics,  then  the  simulation  may  not  mimic  the 
true  system  behavior  adequately.  Once  again,  the  analyst 
can  use  variance  attribution  to  obtain  feedback  regarding 
the  modeling  process.  This  allows  model  refinement  efforts 
to  be  expended  in  accordance  to  factor  sensitivity. 

6  BENEFITS 

The  benefits  of  using  a  robust  design  approach  can  be  sub¬ 
stantial.  Many  of  these  apply  regardless  of  the  type  of  model 
(simulation,  analytic,  prototype)  on  which  experiments  are 
performed.  First,  because  the  chosen  system  configuration 
is  robust,  it  is  likely  to  work  well  across  a  variety  of  real¬ 
izations  of  noise  factor  values.  This  means  that  there  are 
likely  to  be  fewer  unpleasant  surprises  when  the  decision 
is  implemented.  A  second  benefit  is  improved  communica¬ 
tion  between  the  analyst  and  client  via  expected  loss.  This 
also  makes  it  possible  to  evaluate  trade-offs  between  the 
costs  of  reducing  noise  and  the  benefits  of  improving  per¬ 
formance  quality.  There  are  times  when  the  insights  gained 
from  the  robust  design  process  will  allow  decision-makers 
to  simultaneously  improve  performance  and  decrease  costs. 
Finally,  the  fact  that  expected  loss  is  calculated  facilitates 
continuous  improvement:  even  if  target  value  is  achieved 
on  average,  one  can  always  seek  to  identify  factors  that 
can  further  reduce  the  variability  of  the  response.  In  this 
manner,  factors  that  were  initially  classified  as  noise  factors 
may  become  decision  factors  at  a  later  date. 

There  are  some  benefits  of  particular  interest  to  sim¬ 
ulators  as  well.  First,  the  view  that  variability  is  a  criti¬ 
cal  component  of  performance — not  solely  a  nuisance  to 
be  overcome  by  taking  larger  samples — may  improve  the 
modeling  and  analysis  process,  and  lend  more  credibility 
to  the  simulation  results.  Second,  this  structured  approach 
lends  itself  to  rapid  model  evaluation  and  scenario  analysis. 
It  is  much  more  efficient  than  trial-and-error  at  identifying 
“good”  decisions.  Third,  the  variance  attribution  process 
gives  the  simulation  analyst  the  ability  to  test  whether  or 
not  the  model  performance  is  highly  sensitive  to  input  dis¬ 
tributions  and  their  parameters.  This  can  reduce  the  time 
required  to  arrive  at  a  functioning  simulation  model  that 
captures  the  essential  elements  of  the  real-world  process. 
If  certain  input  parameters  have  little  impact  on  the  per¬ 
formance,  then  it  is  not  necessary  to  spend  a  great  deal  of 
time  or  resources  collecting  data  to  build  accurate  empirical 
distributions.  If,  however,  performance  is  found  to  vary 
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greatly  for  small  changes  in  distribution  parameters,  this  is 
critical  to  identify  in  order  to  insure  that  good  decisions  are 
made.  Finally,  one  may  be  able  to  assess  a  priori  whether 
or  not  it  would  be  valuable  to  add  more  complexity  to 
component  models.  This  will  focus  the  modelers’  efforts 
on  refining  the  simulation  in  ways  that  add  value,  rather 
than  simply  adding  to  the  run-time  requirements. 

7  CONCLUDING  REMARKS 

The  approach  outlined  in  this  paper  integrates  the  concepts 
of  robust  design  with  response  surface  metamodeling  and 
system  optimization  efforts  for  discrete-event  simulation. 
The  simulation  arena  is  amenable  to  analysis  using  robust 
design  strategies  since  all  factors  are  controllable  by  the 
analyst.  The  efficiency  gained  by  designed  experimentation 
is  particularly  beneficial  for  complex  simulation  models, 
since  each  realization  of  system  performance  corresponds 
to  the  results  of  a  (potentially  lengthy)  run. 

In  summary,  robust  design  can  be  a  highly  useful  ap¬ 
proach  for  analyzing  models  of  complex  systems  for  several 
reasons. 

•  It  is  flexible.  Robust  design  can  be  applied  to 
terminating  or  non-terminating  simulation  mod¬ 
els  analytic  models,  statistical  models,  or  physical 
prototypes. 

•  It  is  efficient.  Robust  design  can  indicate  when 
model  components  have  sufficient  detail.  Sampling 
plans  can  incorporate  simulation-specific  artificial 
factors,  and  be  chosen  to  keep  either  the  data 
requirements  or  the  analysts’  time  and  effort  low. 

•  Solutions  are  realistic  by  design.  The  suggested 
system  configurations  have  already  shown  that  they 
will  behave  well  over  a  broad  range  of  adverse 
conditions. 

•  It  facilitates  continuous  improvement.  Robust  de¬ 
sign  clearly  indicates  important  determinants  of 
performance  variation,  and  guides  efforts  for  sys¬ 
tem  ‘optimization’  and  improvement  by  conveying 
hidden  costs  to  the  decision-makers. 
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